Cross-Modal Prototype Driven Network for Radiology Report Generation

Radiology report generation (RRG) aims to describe automatically a radiology image with human-like language and could potentially support the work of radiologists, reducing burden manual reporting. Previous approaches often adopt an encoder-decoder architecture focus on single-modal feature learning, while few studies explore cross-modal interaction. Here we propose Cross-modal PROtotype driven NETwork (XPRONET) promote pattern learning exploit it improve task generation. This is achieved by three well-designed, fully differentiable complementary modules: shared prototype matrix record prototypes; network learn prototypes embed information into visual textual features; improved multi-label contrastive loss enable enhance learning. XPRONET obtains substantial improvements IU-Xray MIMIC-CXR benchmarks, where its performance exceeds recent state-of-the-art large margin comparable (The code publicly available at https://github.com/Markin-Wang/XProNet .).